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Abstract 

Richards (1989) has demonstrated that when individuals' reports 
about environments are aggregated to create environmental measures, 
the use of coefficient alpha to estimate single occasion reliability 
can be misleadingly elevated, and that alpha cannot distinguish an 
environmental measure from a "disguised measure of individual 
differences." In this paper I examine the split-sample reliabilities 
of the Effective School Battery teacher psychosocial climate and 
population measures; and I compare these reliabilities with 
school-level alpha coefficients and with one- and two-year retest 
reliabilities previously reported. Split- sample reliabilities are 
usually somewhat smaller than the alpha coefficients calculated at the 
school leve'. for psychosocial climate measures, but they are 
nevertheless generally substantial. Population measures also show 
substantial split-sample reliabilities, implying that teachers are 
clust«red within school environments in ways that are associated with 
their personal characteristics (individual differences). 

Tables showing selected results from an examination of the 
correlates of these school environment and teacher population measures 
with student scholastic performance, attendance, and dropout are 
included to drum up interest in a related research report. 
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The Internal Structure and Correlates of Some School Climate Measures 

As Richards (1978, 1988, 1989) has pointed out, psychologists are 
often confused by an "individual differences" fallacy In assessing the 
psychometric properties of environmental measures. Although 
Information about the homogeneity and factor structure of Items and 
scales examined at the individual level are unlnstructlve about the 
homogeneity and factor structure of environmental summary scores based 
on these items or scales, psychologists and educational researchers 
persist in developing measures and reporting psychometric data for 
environmental or "climate" instruments based on these individual -level 
analyses. For example, individual -level r?sults are presented by Moos 
and Trlckett (1987) for their Classroom Environment Inventory, by 
Proctor and Vlllanova (1984) for their widely used Connecticut School 
Effectiveness Questionnaire, and by many others (Andrews et al., 1987; 
South Carolina State Department of Education, 1987; Halderson, 1987). 

One of the findings in Richards' (1989) most recent contribution 
towards clarifying our thinking on the measurement of environments is 
the compelling demonstration that estimates of the homogeneity of 
environmental measures based on environmental unit means as the basis 
of analysis can be quite misleading. To remind you of his results, 
alpha coefficients for environmental scales- -estimated from 



classroom- level Item means for randomly composed c lass rooms "'vere 
substantial. These alphas approximated the alphas tor scales 
estimated at the individual level. Because his synthetic classrooms 
were randomly composed, the "informants" were providing information in 
response to requests about random classrooms (i.e., their real 
classrooms) , and so the reliability of the measures should have been 
near zero. 

They are not zero because individuals' responses to items are 
correlated at the individual level and the individuals' item responses 
were not independently randomly assigned to classrooms. What this 
finding- -which was surprising both to Richards and to me- -means is 
that alpha coefficients for survey data in which individuals provide 
more than one piece of information about the environment can include 
reliable individual -difference variance in reliability estimates for 
composites scored at the environment level. Alphas calculated at the 
environment level can therefore provide inflated estimates of the 
reliability of environmental measures. 

Richards' (1989) Table 2 shocked me so much that I decided to 
make my planned presentation available to you in printed form 
(Gottfredson & Gottfredson, 1989b; Gottfredson & Stewart, 1989). 
Instead I will try to build on Richards' recent paper and describe 
some further exploration of the reliability of the Effective School 
Battery (Gottfredson, 1984; ESB) using the split- sample technique. 

-3- 



ERIC 



5 



Then, I will take a few minutes to describe some correlates of these 
same ESB scales. 

The ESB Is an Instrument I developed for two main purposes: (a) 
to evaluate school -based programs Intended to ameliorate the risk of 
delinquency, dropout, and school failure, and (b) to provide 
Information that stimulates organization development In schools. The 
scales of the ESB were developed to assess characteristics of program 
Implementation and program outcomes sought by program Implementers 
with whom my colleagues and I have worked over the years. In 
developing these scalds, I sought to devise measures of school climate 
that could be used to supplement measures of academic outcomes with 
measures of the quality of school environments, adolescent 
socialization, and teacher attitudes and practices,^ 

The ESB teacher survey is Intended to measure dimensions of (a) 
school psychosocial climate and (b) teacher population 
characteristics, Conceptually, school climate is a property of the 
school. I regard Individual differences in perceptions of school 
climate as error or "noise," In contrast, a teacher characteristic is 
a property of an individual teacher, and true score variability should 
reflect individual differences within schools. Concepta-^lly , it is an 



^The ESB is composed of both teacher and student surveys. Only the 
teacher surveys are examined here. 



empirical matter whether or not segregation according to individual 
differences or school effects produce variability among schools in 
these population characteristics. 

The nine psychosocial climate scales and the seven population 
characteristic scales of the ESB teacher survey are described in 
Figures 1 and 2, respectively. Details of the development of the ESB 
are reported in the Manual (Gottfredson, 1984) . Briefly, the 
population characteristics scales were based on item analyses 
conducted in the usual way at the individual level. I tried to build 
scales measuring relevant theoretical constructs that measured 
individual differences in personality, attitudes, and self -reported 
behavior. Job satisfaction is an example of such an 
individual -difference variable. I tried to have items call for 
reports about the self ("I like my job" for example). In using these 
population scales to characterize a school's population, individual 
scale scores are averaged. These are useful environmental descriptors 
to the extent that the people make the place (As tin £e Holland, 1961; 
Schneider, 1987). If a program improves teacher job satisfaction, 
then the Job Satisfaction mean score should go up. The standard error 
of the mean summarizes the efficiency of the estimate of this mean. 

In contrast, I tried to use candidates for measures of 
psychosocial climate that call for reports about the place ("The 
student government makes important decisions"). Scales were 
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constructed based on Item analyses at the school level of analysis. 
That is, item responses were aggregated to the school level and then 
their correlations were examined. In scoring scales, items are 
aggregated to produce school means and then these means are summed to 
produce scores. ^ This approach was intended to select items that 
measured differences among schools rather than differences among 
teachers. I then examined scale correlations and reliabilities at the 
school level using such techniques as coefficient alpha. 

Richards' recent paper is extremely important because it means 
that despite the school -level item analysis and despite the attempt to 
avoid personality- like items (which, by the way, I found difficult), 
the alpha reliabilities I have reported could be misleading. 

Fortunately, I happened to have data for 68 schools which had 
recently been assessed using the ESB by my dosk as I read Richards' 
(1989) paper. These are all the schools in a county-wide school 
district, and they were assessed in 1988 as part of an action research 
project (Gottfredson & Gottfredson, 1989a). 

The corrected split-sample reliability coefficient (which I 
propose we henceforth call the Richards Cv fficient) was computed by 



Incidentally, the slight variations in observed setting means in 
Richards' (1989) Table 1 probably result because a mean of means when 
groups differ in size is not generally equal to the simple mean across 
all groups. 
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assigning even numbered teachers in each school to one half- sample and 
odd numbered teachers to a second half-sample. Then scores were 
computed for each half- sample within each school as I described 
earlier. Population scores for each half are the mean of the 
individual -difference scores for that half; psychosocial climate 
scores for each half are means of the item means for that half. 
Split-sample means were correlated and corrected using the 
Spearman- Brown formula. 

The results are shown in the first column of Table 1. In 
general, the split-sample reliabilities are moderate (.62) to high 
(.88) except for the scale labeled "Avoidance of the Use of Grades as 
a Sanction." Interestingly, this composite is formed from two "I" 
items, it has not always fared well in earlier examinations of its 
properties (Gottfredson, 1984), and practical experience implies that 
its interpretation and use is cumbersome and difficult. 

In general, the split-sample reliabilities for the individual 
difference measures are lower than the psychosocial climate measures. 
Most of them are far from zero, however. This implies that people are 
clustered within schools in a manner that is not independent of their 
attitudes and behaviors. Were this not true, the coefficients would 
be near zero as they were for the randomly composed "classrooms" in 
Richards' Table 2. Notice that the coefficient for Professional 
Development is highest and that it is likely that if one teacher in a 
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school is exposed to a staff development experience it is likely that 
most would be exposed. The split-sample coefficient for 
Nonauthoritarian Attitude is lowest, and this seems closest to a pure 
personality measure. 

The second column in Table 1 summarizes the homogeneity 
coefficients I reported in the ESB Manual. In most cases the alphas 
are higher than the Richards coefficients. The alphas are from 
different samples, and some differences would be expected for this 
reason. Nevertheless, the pattern in results suggests that the 
school- level alphas tend to overestimate the reliability. 

The remaining two columns show data I have reported previously 
about the retest reliability of these scales. It may be useful to 
reflect what is treated as error in these coefficients and what is not 
treated as error. Unsystematic variation over time is treated as 
error, but these re- test reliabilities for the psychosocial cllmaf.e 

do not necessarily exclude all individual differences variance. 
To the extent that the inhabitants of the environment do not all turn 
over during the time interval ^ that item responses at time 1 are 
correlated with item responses at time 2 at the individual level and 
that people are clustered within environments in the basis of these 
stable individual differences or on the basis of a characteristic that 
is correlated with them, these re-test reliabilities can be inflated. 
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Of course the retest coefficients for the population 
characteristics do Include Individual differences variance. They are 
Intended to. In theory, one can Increase the average job satisfaction 
In an environment by replacing unhappy people with happy people, for 
example. To the extent to which people are clustered within 
environments on the basis of the measvired characteristics or on the 
basis of a characteristic correlated with a measured 
characteristic- -for whatever reason- -then the Richards coefficients 
will be non-zero. And to the extent that they stay there the retest 
coefficients will be non-zero. Among the reasons for clustering could 
be differential assignment to environments, differential attrition 
from environments, segregation of one kind or another, self-selection 
or choice of environments, or something the environments do to the 
people (i.e., environmental effects). 

How would one improve on the retest estimate for a psychosocial 
climate measure? Split the populations of time 1 inhabitants into 
random halves and identify members of each half. Randomly assign 
newcomers to the environment at time 2 to half-1 or half -2, and assign 
perslsters co the same half they were assigned at time 1. Correlate 
the time-l-half -1 scores with the time-2-half-2 scores and the 
time-l-half -2 scores with the time-2-half -1 scores. Then step up the 
mean of these correlations. I am sorry to report that I have not had 
the time to do this for you. 
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There are some more tables I want to show you briefly to try to 
get you liiterested In the paper I had planned to give today 
(Gott£redson & Gottfredson, 1989b), which you can pick up from the 
front table. Table 2 shows correlations between the psychosocial 
climate scales we have Just been discussing and some academic 
ac^ levement outcomes for elementary school children. All those stars 
In the table are dazzling, but what this table does not reveal is that 
both ESB scores and scholastic performances aggregates are efficiently 
predicted by the affluence and ethnic composition of schools. Table 3 
shows that net of these characteristics the correlations tend to be 
much smaller* Whiz -bang correlations in highly aggregated data are 
not hard to come by but they can be hard to interpret (Gottfredson, 
1979), 

Table 4 shows correlations between the ESB scales we have been 
discussing and school attendance, and Table 5 shows correlations with 
dropout rates. 
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Table 1 

Single Occasion and Re-Test Reliabilities of Two Kinds of Environmental 
Assessments Contained in ti.e ESB 



Corrected School -level One -year Two-year 
ESB scale split-sample alpha retest retest 



Psychosocial climate 



Safety 


.81 


.94 


.75 


.51 


Morale 


.80 


.92 


.62 


.65 


Planning & action 


.69 


.88 


.51 


.33 


Smooth administration 


.78 


.92 


.56 


.55. 


Resources 


.81 


.84 


.72 


.76 


Race relations 


.77 


.76 


.69 


.60 


Parent/community involvement 


.88 


80 






Student infTubr.ce 


.62 


.83 


.59 


.76 


Avoidance of grades as sanction 


.38 


.74 


.45 


.28* 


Population characteristic 










Pro integration attitude 


.55 


NA 


.39 


.64 


Job satisfaction 


.49 


NA 


.66 


.41 


Interaction with students 


.59 


NA 


.60 


.71 


Personal security 


.66 


NA 


.36 


.19* 


Clasroom orderliness 


.53 


NA 


.46 


.43 


Professional development 


.75 


NA 


.47 


.62 


Nonauthoritarian attitude 


.22 


NA 


.31 


.47 


Number of schools 


(68) 


(52-65) 


(46-52) 


(33-38) 


* Not significantly different from zero. 
NA - not applicable. The scores are means 


of individual 


scores . 





Note. Corrected split half reliabilities are from the population of 68 
schools in a county-wide school district. School level alphas are 
calculated from school-level (aggregated) item means, and are the medians 
of two alphas (52 to 65 schools) reported for two diverse samples of 
schools in the ESB Manual. The re -test reliabilities are from a diverse 
sample of schools that were assessed one and two years apart, and are taken 
from the ESB Manual. 
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Table 2 



Correlations Between Student Characteristics or Educational Outcomes 
and the Teacher Climate Scales of the Effective School Battery: Grade 3 
(N - 41 to 42 Schools) 



Effective School Battery Score 

Schools' Student Population 

Characteristics or Planning Smooth Race Prnt/Cmty Student Avof.d. Grd. 

Educational Outcomes Safety Morale & Action Admin. Resources Relations Involv. Infl. Sanction 



Mean student age 


-.55*** 


-.37* 


-.40** 


-.27 


. .48*** 


..53*** 


-.62*** 


-.25 


-.20 


Percentage never retained 


.47** 


.44** 


.48*** 


.36* 


.52*** 


.48*** 


.68*** 


.32* 


.14 


Percentage white 


.48*** 


.35* 


.25 


.18 


.42** 


.42** 


.62*** 


-.02 


.24 


Percentage free/reduced lunch 


- . 58*** 


-.47** 


-.35* 


-.26 


-.48*** 


-.46** 


-.75*** 


-.13 


-.21 


Percentage nale 


-.23 




-.26 


-.14 


-.14 


-.10 


-.14 


-.06 


-.05 


Mean reading score 


.61*** 


.50** 


.38* 


.19 


.40** 


.46** 


.73*** 


.26 


.19 


Mean nath score 


. 68*** 


.52** 


.43** 


.18 


.40** 


.44** 


. 70*** 


.34* 


.27 


t meeting reading criterion 


. 54*** 


.34* 


.18 


.04 


.26* 


.27 


.47** 


.14 


.21 


% meeting math criterion 


. 69*** 


.52** 


.35* 


.21 


.40* 


.39* 


. 60*** 


.29 


.41** 


% meeting math criterion on tine 


. 64*** 


. 60*** 


.58*** 


.37* 


.54*** 


. 54*** 


.78*** 


.40** 


.26 


% meeting reading crit. on time 


.56*** 


.52** 


. 54*** 


.31 


.51** 


.55*** 


. 74*** 


.39* 


.21 


Mean reading, on- time students 


.56*** 


.48** 


.36* 


.16 


.36* 


.50** 


. 69*** 


.21 


.14 


Mean math, on- time students 


.66*** 


.52*** 


.41** 


.20 


.37* 


. 53*** 


.69*** 


.28 


.18 



* p < .05 
** p < .01 
'*** p < .001 
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Table 3 



Partial Corralatlona of Effactive School Battery Scalea with Educational Outcomes 
Controlling for Percentage Black and Percentage of Students Receiving Free or Reduced 
Lunch: Grade 3 (N « 41 Schools) 



% meeting cri« Mean score i 
Mean score: t meeting criterion terion on time: onetime students: 

ESB 

Scale reading math reading math math reading reading math 



Safety 


.27 


.46** 


.28 


.52** 


.36* 


.21 


.20 


.44** 


Horalc 


.20 


.25 


.04 


.32* 


.39* 


.26 


.17 


.26 


Planning & 


.14 


.23 


-.08 


.17 


.48** 


.41** 


.10 


,20 


Ac don 


















Snooch 


-.07 


-.05 


-.19 


.06 


.25 


.15 


-.11 


-.02 


Adnlnif tr aclon 


















Raaourcas 


.01 


.10 


-.06 


.15 


.29 


.26 


-.04 


.05 


Raca 


.18 


.21 


.00 


.17 


.35* 


.37* 


.28 


.36* 


Ralatlona 


















Parant/C&ty. 


.27 


.33* 


-.01 


.26 


.48** 


.38* 


.22 


.32* 


Involvamant 


















Scudane 


.18 


.24 


.00 


.27 


.42** 


.38* 


.10 


.15 


Influanea 


















Avoid. Gradta 


.07 


.23 


.14 


.38* 


.18 


.11 


-.01 


.10 


Sanceion 


















Pro-Incagraelon 


.23 


.38* 


-.04 


.09 


.41** 


.47** 


.14 


.33* 


Ateicuda 


















Job 


-.04 


.06 


-.05 


.23 


.30 


.27 


-.06 


-.05 


Satlffaeeion 


















Intaracelon 


-.12 


-.06 


-.05 


-.10 


-.28 


-.20 


-.17 


-.23 


with SCudanCs 


















Parsonal 


.49** 


.52** 


.24 


.51** 


.54** 


• ^2** 


.47** 


. 52** 


Saouriey 


















ClaairooBi 


.24 


.43** 


.18 


.55** 


.33* 


.20 


.24 


.30 


Ordarlinasa 


















Profaffional 


.05 


.10 


.02 


.26 


.36* 


.26 


-.04 


-.01 


Davalopment 


















Nonauchor 1 ear Ian 


.31 


.36* 


.15 


.39* 


.46** 


.38* 


.24 


.32 


Ateicuda 



















* p < .05 
** p < .01 
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Correlationg of Effective School Battery Scales with Student Att«nH.«... 
Elementary. Middle. And ui^h c-k^^i- ^ &tuaent Attendance: 



Elementary, Middle, and High Schools 

Zero order correlations 



ESB 

Scale 


Elementary 
(N - 42) 


Middle 

(N - 15) 


High 

(N - 11) 


Total 
(N - 68) 


c* X e laon u a ry 
(N - 42) 


Middle & High 


Safety 


.55*** 


.55* 


.29 


. 53*** 


1 J J n W W 




Morale 


.51*** 


.48 


•.34 


.42***^ 






Planning & 
Action 


.36* 


.32 


.43 


. 43*** 


t AO 




Smooth 
AoiBinittratlon 


.29 


.08 


-.13 


.20 


. 17 


no 


Resources 


.24 


.40 


-.34 


.12 


.14 


< WW 


Race 
Keiacions 


.39* 


.24 


.25 


.32** 


.36* 


• 


Parene/Cfflty. 


.40** 


.28 


.55 


.40*** 




.26 












.27 


Student 


.38* 


.36 


.40 


.43*** 


.27 


.33 


Avoid. Grades 
AS Sanction 


.26 


.32 


.51 


.45*** 


.35* 


.49* 


Pro -Integration 
Attitude 


-.08 


.21. 


.18 


.11 


.00 


.20 


Job Satis* 
faction 


.10 


.45 


.33 


.20 


-.08 


.36 


Interaction 
with Students 


.00 


-.22 


.16 


• . 42*** 


-.19 


-.12 


Personal 
Security 


.51*** 


.46 


.04 


. 44*** 


.47** 


.27 


Classroom 
Orderliness 


.29 


.63* 


.42 


.08 


.22 


.Six 


Professional 
Development 


.25 


.24 


.47 


.38*** 


.20 


.37 


Nonauthorltarlan 
Attitude 


.38 


.14 


.34 


.36** 


.42** 


.10 



* p < .05 ** p < .01 *** p < .001 
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Table 5 



Corr«l*tlons of Effectivd School B«tt*ry Teacher Psychosocial Climace 
Scales wlch Oropouc Race: Middle and High Schools 



Zero order correlations Partial r 



ESB Middle High Total Total 

Sc*l« <N - 15) (N - 11) (N - 26) (N - 26) 



Safety 


. , 78*** 


-.20 


. .61*** 




Morale 


- . 64** 


-.44 


. . 61*** 


• .48 


Planning and Action 


-.18 


-.16 


-.42* 


-.13 


Saooth Adalnlstratlon 


.09 


-.35 


-.38 


•.16 


Resourcea 


-.45 


.03 


- .53** 


- 26 


Race Relatione 


-.04 


.06 


-.06 


-.19 


Parent/Com&unlty Involvement 


-.50 


.17 


-.38 


-.10 


Student Influence 


-.32 


.09 


-.24 


-.18 


Avoidance of Grades 


-.57* 


-.50 


-.57** 


-.69** 


as Sanction 






Pro 'Integration Attitude 


.25 


.35 


.02 


.14 


Job Satisfaction 


-.46 


-.14 


-.47* 


-.37 


Interaction with Students 


.08 


-.23 


.48* 


.14 


Personal Security 


-.44 


-.37 


-.31 


-.33 


Classrooa Orderliness 


• .66** 


-.05 


-.13 


-.51 


Professional Development 


-.23 


.46 


-.29 


- .44 


Nonauthorltarlan Attitude 


-.18 


-.26 


-.30 


.13 



Not0. Dropout rates are annual Incidence rates for grades 7 and 
8 for middle schools and for grades 9 • 12 for high schools. 

•Controlling for percentage white, percentage free or reduced 
lunch, and. school level. 

* p < .05 ** p < .01 *** p < .001 
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Figure 1 

ESB School Psychosocial Climate Scales- -Teacher Reports 



Scale (No. of Items) 



Meaning 



Safety (10) 



Morale (11) 



Planning and 
Action (9) 



Smooth 

Administration (12) 



Resources (4) 



Race Relations (2) 



Parent/Community 
Involvement (6) 

Student 
Influence (5) 

Avoidance of Use of 
Grades as a Sanction (2) 



Indicates how safe teachers report the school 
environment to be. A high score means that 
teachers tend to report most places in the school 
to be safe, and a low score means that teachers 
report many places in the school to be unsafe. 

Indicates the degree of enthusiasm of a school's 
faculty and faculty confidence in the school. A 
high score means that teachers are likely to be 
enthusiastic and to participate in the development 
of new programs. A low score suggests that many 
faculty share a sense of resignation about the 
school and little confidence that much can be done 
about it. 

Indicates teacher reports of the degree to which 
the school takes an experimenting or innovative 
approach to planning school programs. 

Indicates how teachers perceive the school 
administration. High scores imply that teachers 
perceive that they get thu help they need to do 
their jobs when they need it. 

Indicates whether teachers report adequate 
instructional supplies and other resources or 
whether they report difficulty in obtaining needed 
teaching supplies. 

Indicates (in integrated schools) how well 
different ethnic groups get along. In schools with 
students and faculty of only one ethnic group, this 
scale should be disregarded. 

Indicates the degree to which the school uses 
community resources in its programs. 

Indicates teacher perceptions of the extent to 
which students participate in school decisions. 

Indicates the extent to which teachers avoid 
lowering grades in response to student misconduct-- 
a generally poor practice. 
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Figure 2 

ESB Teacher Population Characteristics 



Scale (No. of Items) 



Meaning 



Pro- Integration 
Attitude (4) 



Job 

Satisfaction (3) 



Interaction 
with 

Students (6) 



Personal 
Security (8) 



Indicates average teacher attitude toward 
Integrated education. A high score suggests that 
teachers view integrated education In a positive 
way; a low score suggests that the average teacher 
may be somewhat Insensitive to Issues of racial 
equity. 

Indicates how the average teacher feels about his 
or her job. A high score Implies that teachers 
typically like their jobs In the school; a low 
score Indicates that teachers typically dislike 
their jobs. 

Indicates how much positive social interaction the 
average teacher reports having with students. A 
high score implies that many teachers report 
friendly Interaction with students. 

Indicates the average teacher's experience of 
personal victimization. In a low -scoring school, 
relatively many teachers report receiving obscene 
remarks or gestures, threats, thefts, or even 
attacks. A high score Implies teachers rarely 
experience indignities or victimization in the 
school. 



Classroom 
Orderliness (2) 



Indicates how orderly the average teacher's 
classroom is. A high score Implies classrooms are 
typically orderly; a low score implies that 
disruption Interferes with teaching in many 
classes. 



Professional 
Development (8) 



Nonauthoritarlan 
Attitude (3) 



Indicates how much exposure to continuing education 
the average teacher in the school has had in the 
past year. 

Indicates the average teacher's attitude about 
student- teacher authority relations. A low score 
implies many teachers have a punitive, moralistic 
attitude about student misbehavior. A high score 
implies many teachers have a more flexible attitude 
about coping with student misconduct. 
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